Conceptualization and Measurement

PSCI 3300.002 Political Science Research Methods

A. Jordan Nafa

University of North Texas

2/28/23

Overview

  • Social science is about developing and evaluating causal theories

    • Does economic development lead to democratization?

    • Do gender quotas cause an increase in female representation?

  • Theories are comprised of linkages between concepts

    • Gender Quotas, electoral institutions, women’s CSOs, public opinion, and political corruption

    • Causal relationships are comprised of concepts that influence one another

    • Important to consider how we measure these concepts

\[ \definecolor{treat}{RGB}{27,208,213} \definecolor{outcome}{RGB}{98,252,107} \definecolor{baseconf}{RGB}{244,199,58} \definecolor{covariates}{RGB}{178,26,1} \definecolor{index}{RGB}{37,236,167} \definecolor{timeid}{RGB}{244,101,22} \definecolor{mu}{RGB}{71,119,239} \definecolor{sigma}{RGB}{219,58,7} \newcommand{normalcolor}{\color{white}} \newcommand{treat}[1]{\color{treat} #1 \normalcolor} \newcommand{resp}[1]{\color{outcome} #1 \normalcolor} \newcommand{sample}[1]{\color{baseconf} #1 \normalcolor} \newcommand{covar}[1]{\color{covariates} #1 \normalcolor} \newcommand{obs}[1]{\color{index} #1 \normalcolor} \newcommand{tim}[1]{\color{timeid} #1 \normalcolor} \newcommand{mean}[1]{\color{mu} #1 \normalcolor} \newcommand{vari}[1]{\color{sigma} #1 \normalcolor} \]

Concepts and Measurement

  • Concepts are theoretical abstractions

    • Sometimes measurement is straightforward

      • What is your age?

      • What state do you live in?

    • Other times measurement is more complicated

      • What is political corruption?

      • What is female representation?

      • What is democracy?

    • Have to develop an operational definition of a concept before we can measure it

Description

Description has a valuable place in political science and accurate description is essential to empirical research.

  • Descriptive research is comprised of three primary goals

    • To answer research questions

    • To generate research questions

    • To do both of these iteratively

  • Yet, it is necessarily inferior to causal approaches because it cannot answer questions of why or how things happen

Descriptive and Causal Questions

  • Descriptive Question: Who Votes for Trump?

  • Causal Question: Why do people vote for Trump?

  • Descriptive Question: What is media bias?

  • Causal Question: Does media bias influence political attitudes?

Quantitative Description

  • A rectangular, observation-by-variable dataset

    • “dataset observations” (DSOs)
  • A clear unit of analysis

    • Country-years, state-months, respondent-year, etc.
  • Requires multiple cases/units

    • Nothing can be learned from an \(\sample{n}\) of 1
  • Quantitative and qualitative measures

    • In the social sciences quantitative variables are measures of qualitative concepts
  • Calculation of summary statistics

Tabular Data

country_name year corruption quota_scale female_leg femaleleg_delta elections
Mexico 1979 8.21 No Quota 7.80 -0.20 0
Mexico 1982 7.86 No Quota 11.30 3.50 0
Mexico 1985 7.73 No Quota 11.00 -0.30 0
Mexico 1988 7.73 No Quota 12.00 1.00 0
Mexico 1991 7.70 No Quota 7.60 -4.40 0
Mexico 1994 6.79 No Quota 14.20 6.60 0
Mexico 1997 6.46 No Quota 14.20 0.00 0
Mexico 2000 5.52 No Quota 16.00 1.80 0
Mexico 2003 6.11 Quota, Strong Sanctions 22.60 6.60 1
Mexico 2006 6.10 Quota, Strong Sanctions 22.60 0.00 2
Mexico 2009 6.10 Quota, Strong Sanctions 28.20 5.60 3
Mexico 2012 6.62 Quota, Strong Sanctions 36.80 8.60 4
Mexico 2015 7.42 Quota, Strong Sanctions 42.37 4.97 5
Mexico 2018 6.89 Quota, Strong Sanctions 48.20 5.60 6
Mexico 2021 5.04 Quota, Strong Sanctions 50.10 1.90 7

Operationalization

  • Measure features

    • Need to consider level of measurement and how to score each case on each feature

    • To study concepts, we need to be able to observe those concepts and encode them as variables

    • Can’t study things we can’t measure and can’t measure things we can’t observe

      • Exception to this is latent variables that we can’t observe but can try to measure
  • The definition of a variable

    • A dimension that describes an observation or the operationalization of a concept

Measuring Democracy

  • How should we measure democracy?

    • Requires we first define democracy

    • One of the oldest debates in comparative politics

    • Sometimes the worst, except for all the others is the best we can do

    • As a general rule, treating complex multi-dimensional concepts as dichotomies is at best logically fallacious

    • Need to make principled tradeoffs between parsimony and precision

Assessing Measurement Quality

  • Conceptual Clarity

    • Conceptual Clarity is about knowing what we want to measure

    • Poorly defined concepts lead to poorly measured variables

      • Ambiguity

      • Vagueness

  • Revise concept definition as needed

    • But revisions have to made before your analysis

Assessing Measurement Quality

  • Construct Validity

    • Construct Validity is the degree to which a variable measures a concept

    • Does our variable actually measure what we say it measures or some secret third thing?

    • Construct validity is high if a variable adequately measures the concept we care about

    • Construct validity is low if a variable is actually a measure of something else

Example: Polity and Democracy

  • Common measure of democracy that was dominant in IR and Comparative Politics for several decades

  • Mostly a measure of institutional constraints on the executive

  • Are institutional constraints on the executive and democracy the same thing?

    • There are people who think stupid things, some of them in this department, who would argue yes

    • As we established earlier, these people are wrong

  • Polity is the QWERTY keyboard of political science

    • An inferior product that became entrenched by being first to market

Why Does Measurement Matter?

  • The measures we use often influence the answers we get

  • As a general principle, the result of any operation performed on a precise number and garbage is some variation of garbage

    • \(\mathrm{Precise~Number} \times \mathrm{Garbage} = \mathrm{Garbage}\)

    • \(\sqrt{\mathrm{Garbage}} = \mathrm{Less~Bad~Garbage}\)

    • \(\mathrm{Garbage}^{2} = \mathrm{Worse~Garbage}\)

    • \(\frac{1}{\sample{n}}\sum_{\obs{i}=\obs{1}}^{\sample{n}}[ \mathrm{Garbage}_{\obs{i}}] = \mathrm{Better~Garbage}\)

    • \(\mathrm{Precise~Number}^{\mathrm{Garbage}} = \mathrm{Much~Worse~Garbage}\)

    • \(\mathrm{E[\resp{Y} _{\obs{i}} = Garbage~|~\treat{X}_{\obs{i}}=Precise~Number}] = \mathrm{Expectation~of~Garbage}\)

    • \(\resp{y} _{\obs{i}} = \alpha + \hat{\beta_{1}}\times\mathrm{Polity~IV}_{\obs{i}\tim{t}} = \mathrm{Comparative~Garbage}\)

Assessing Construct Validity

  • Most thick concepts are multidimensional

  • Things look for

    • Convergence (Convergent Validity)

    • Discrimination (Discriminant Validity)

    • For example, the multi-trait, multi-method matrix

Proxies and Partial Measures

Why do we care?

  • Once we have measured variables for observations, we can conduct analysis!

    • And not a damn moment sooner
  • And once we have analysis, we can draw inferences and make evidence-based claims

    • Political science is a field interested in evaluating causal theories.

    • No one care what you think unless you back it up with empircally valid evidence

Summary

  • We want to make claims about concepts

  • But we only observe and can only analyse observed, measured variables

  • So our task as social scientists is to

    • Link the concepts we care about to observable phenomena

      • Components, indicators, variables

      • Draw out theoretical implications from measures

  • Good measurement is theory-driven

References

Coppedge, Michael, Staffan Lindberg, Svend-Erik Skaaning, and Jan Teorell. 2016. “Measuring High Level Democratic Principles Using the v-Dem Data.” International Political Science Review 37: 580–93.
Lindberg, Staffan I., Michael Coppedge, John Gerring, and Jan Teorell. 2014. “V-Dem: A New Way to Measure Democracy.” Journal of Democracy 25: 159–69.